Uncovering cognitive processes: cued retrospective reporting based on eye-movement records

The verbal reporting techniques used most in research on learning and instruction, concurrent and retrospective reporting, both have drawbacks. Retrospective reporting often results in omissions/fabrications, and concurrent reporting is difficult to implement when tasks impose high cognitive load or contain auditory information. Cued retrospective reporting (CRR) based on eye-movement records might be able to overcome these drawbacks: while maintaining the retrospective nature, the cue shows both physical (mouse/keyboard) and cognitive (eye movements) actions, thereby presumably leading to less omissions/fabrications. Because a previous study showed promising results, the present study extends the test of CRR to qualitative data (i.e., process coverage). Verbal reporting techniques are used in a variety of disciplines to study cognitive processes, such as problem-solving, user-system interaction, or decision-making processes. In the field of instructional design, for example, attempts to uncover cognitive processes of experts are made to obtain input for the design of instructional materials, and cognitive processes of learners are uncovered while they interact with instructional materials to evaluate their effectiveness. The two most widely used verbal reporting techniques are concurrent and retrospective reporting. Concurrent reporting (Ericsson & Simon, 1993; Van Someren, Barnard, & Sandberg, 1994) is an online technique, that is, it requires participants to verbalize all thoughts that come to mind during task performance. Retrospective reporting (Ericsson & Simon, 1993) on the other hand, is an offline technique, and requires participants to report the thoughts they had while they were working on a task immediately after task performance. Both techniques allow for valid inferences about the cognitive processes underlying task performance when the verbalization instructions and prompts are worded in such a way that the evoked responses do not interfere with the cognitive processes (e.g., participants should not be asked to explain or elaborate; see Ericsson & Simon, 1993, for information on appropriate instructions and prompts). However, both techniques have drawbacks. Because concurrent reporting is done during task performance, it is difficult to maintain under conditions of high cognitive load (Ericsson & Simon, 1993; Van Gog, 2006). For novices, even a task that might seem relatively simple (e.g., a math problem with only one possible solution) can impose a high load on working memory when it contains many new information elements that interact with each other (i.e., intrinsic cognitive load; Sweller, Van Merriënboer, & Paas, 1998). In addition, concurrent reporting cannot be used with tasks that contain auditory information, because reporting will interfere with processing the auditory information. A drawback of retrospective reporting, is that on longer tasks, there is a serious risk that information is omitted or fabricated. Van Gog, Paas, Van Merriënboer, and Witte (2005) proposed that cued retrospective reporting, in which a retrospective report is cued by a replay of a record of eye movements (which also includes mouse and keyboard operations), might be able to overcome these drawbacks of concurrent and retrospective reporting. The retrospective nature would be maintained, thereby imposing less demands on working memory than concurrent reporting. Due to the cue, less omissions or fabrications of actions would be expected than in retrospective reporting without a cue. They chose this cue rather than a video for example, because records of eye movements also include mouse and keyboard operations. Like with a video, the mouse and keyboard operations could stimulate reporting of thoughts regarding cognitive processes that resulted in physical actions. However, some cognitive processes are purely cognitive, and never result in mouse or keyboard actions. For example, when searching for information on the internet, a video or mouse/keyboard record would show only which websites were opened. It would not show whether links to other websites were observed but not opened. Here lies the added value of eye movements in the cue, because eye movements reflect visual attention allocation (Duchowski, 2003; Rayner, 1998), they could trigger reporting of thoughts regarding purely cognitive actions (cf. Hansen, 1991). Van Gog et al. (2005) found that both concurrent and cued retrospective reporting resulted in more action information, as well as in more strategic and conditional information than retrospective reporting without a cue, and found no significant differences between concurrent and cued retrospective reporting. Given these results, it is safe to conclude that retrospective reporting without a cue is not the most optimal method on tasks of longer duration (i.e., more than 2 minutes). However, it is more difficult to decide whether concurrent and cued retrospective reporting are indeed equal, because Van Gog et al. analyzed at the level of frequencies of utterances. Since participants might talk more in “whole sentences” in cued retrospective reporting while they might make several remarks about a single action in concurrent reporting, their data cannot show whether the methods differ in how much of the task performance process they cover. Hence, the present study compares the information elicited with the techniques of concurrent and cued retrospective reporting. It is hypothesized that cued retrospective reporting covers the process better than, or equal to concurrent reporting. Note that both outcomes would be desirable, because in case of equality, cued retrospective reporting could provide researchers with a good alternative when concurrent reporting is not feasible due to cognitive load or task characteristics. Short description of the study Participants were 20 psychology students, novices to the topics of the information search tasks used here (on the (un-)reliability of memory and the (non-)existence of altruism). Participants always engaged in cued retrospective reporting on the first task, and in concurrent reporting on the second, but the task content was counterbalanced, that is, there were two sequences (memory-altruism and altruism-memory) to which participants were randomly assigned. Both tasks consisted of a Google search hit list, from which participants had to select five web-pages that they felt contained useful information for writing a short article in a popular psychology magazine (i.e., they did not actually have to write the article, they only had to judge and select information from the hit list). The hit list and all the web-pages on it, were especially constructed for the experiment so that all participants had access to the same information and only that information. Participants’ eye movements were recorded on both tasks with the remote 50 Hz Tobii 1750 eye tracker, which is integrated with a PC screen, operating on ClearView software (see http://www.tobii.com). The web stimulus recording mode was used, so not only the eye movements, but the entire task performance process (including mouse and keyboard operations) was captured. Participants’ verbalizations were recorded digitally using Audacitiy 1.2.6 (http://audacity.sourceforge.net) with an external microphone attached to the stimulus PC. After the experiment, an “action protocol” was made for each task based on the replays of the eye movement records, describing participants’ actions (e.g., clicks on hit X). The verbal protocols were then matched to those action protocols. Data are currently being analyzed, and results will be presented during the fire hose session.